High Performance Large Scale Web Spider Architecture
نویسندگان
چکیده
This paper describes a cluster-based high-performance web spider architecture. Its architecture has been designed for handling a very large number of web pages with both URLs contents compression. The method we used to fetch URLs has been designed for achieving maximum performance with respect to well-known spider’s considerations. In experiments, our spider achieves an average download rate of 618 URLs/sec and 6 MBytes/sec.
منابع مشابه
Linking Native and Invader Traits Explains Native Spider Population Responses to Plant Invasion.
Theoretically, the functional traits of native species should determine how natives respond to invader-driven changes. To explore this idea, we simulated a large-scale plant invasion using dead spotted knapweed (Centaurea stoebe) stems to determine if native spiders' web-building behaviors could explain differences in spider population responses to structural changes arising from C. stoebe inva...
متن کاملHigh-performance spider webs: integrating biomechanics, ecology and behaviour.
Spider silks exhibit remarkable properties, surpassing most natural and synthetic materials in both strength and toughness. Orb-web spider dragline silk is the focus of intense research by material scientists attempting to mimic these naturally produced fibres. However, biomechanical research on spider silks is often removed from the context of web ecology and spider foraging behaviour. Similar...
متن کاملLessons Learned in Deploying the World’s Largest Scale Lustre File System
The Spider system at the Oak Ridge National Laboratory’s Leadership Computing Facility (OLCF) is the world’s largest scale Lustre parallel file system. Envisioned as a shared parallel file system capable of delivering both the bandwidth and capacity requirements of the OLCF’s diverse computational environment, the project had a number of ambitious goals. To support the workloads of the OLCF’s d...
متن کاملBehavioural and biomaterial coevolution in spider orb webs.
Mechanical performance of biological structures, such as tendons, byssal threads, muscles, and spider webs, is determined by a complex interplay between material quality (intrinsic material properties, larger scale morphology) and proximate behaviour. Spider orb webs are a system in which fibrous biomaterials--silks--are arranged in a complex design resulting from stereotypical behavioural patt...
متن کاملSemantic Constraint and QoS-Aware Large-Scale Web Service Composition
Service-oriented architecture facilitates the running time of interactions by using business integration on the networks. Currently, web services are considered as the best option to provide Internet services. Due to an increasing number of Web users and the complexity of users’ queries, simple and atomic services are not able to meet the needs of users; and to provide complex services, it requ...
متن کامل